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Introduction 


Research on teaching and learning has led to the development of instructional methods that 
are more effective in helping students learn. These evidence-based teaching practices, which 
encourage student engagement and active learning, as well as create inclusive learning en- 
vironments, have been documented by high-level reports and policy papers. At the same 
time, the national policy environment is reflecting a more coordinated effort to improve un- 
dergraduate science, technology, engineering and mathematics (STEM) education across 
relevant organizations and actors. Several multi-institutional projects to improve undergrad- 
uate STEM education are underway, along with an even greater number of institutional-based 
projects aimed at addressing teaching and learning in STEM fields. Documenting the orga- 
nizational change process and impact within and across these projects, developing mech- 
anisms for sharing knowledge, and broadening the group of institutions engaged in this kind 
of reform is essential for realizing large-scale improvements in STEM education in colleges 
and universities across the country. 


The National Science Foundation (NSF) plays an important leadership role in the develop- 
ment and implementation of efforts to enhance and improve STEM education in the United 
States. Through the Education and Human Resources, Division of Undergraduate Educa- 
tion’s Improving Undergraduate STEM Education (IUSE) program, the NSF has made a sub- 
stantial commitment to improve the quality and effectiveness of undergraduate STEM edu- 
cation for all students. 


During a 2019 workshop, the American Association of Universities (AAU) convened experts 
involved in leading, researching, evaluating and funding efforts to transform undergraduate 
STEM teaching across all sectors of higher education. The goal was to synthesize essential 
issues and challenges in designing, leading, and researching institutional and multi-institu- 
tional transformation projects to improve undergraduate STEM education. 


The workshop was designed to identify critical themes necessary to understand when ex- 
amining the outcomes of a reform effort, and when examining the change process of a re- 
form effort. In addition, the workshop created a forum to provide recommendations for the 
continued advancement of the IUSE program through the NSF’s Education and Human Re- 
sources, Division of Undergraduate Education. 


This workshop report serves as an executive summary of key findings and recommendations 
that emerged from an analysis of small-group activities and notes from whole-group discus- 
sions. We extend our thanks to the workshop participants. We also want to extend our ap- 
preciation to the academic leaders and scholars who wrote reflections to ensure the voices 
and perspectives of all sectors of higher education were represented in the summary report. 
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Background on the NSF 
Improving Undergraduate 
STEM Education Program 


The NSF plays a leadership role in the develooment and implementation of efforts to enhance 
and improve STEM education in the United States. Through the Education and Human 
Resources, Division of Undergraduate Education’s IUSE program, the NSF has made a 
substantial commitment to the highest caliber undergraduate STEM education through an 
agencywide framework of investments. The goal of the IUSE program is to improve the 
effectiveness of undergraduate STEM education for both majors and nonmajors, educate 
students to be leaders and innovators in emerging and rapidly changing STEM fields, and 
educate a scientifically literate populace. 


Within the IUSE program are two tracks: (1) Engage Student Learning and (2) Institutional and 
Community Transformation. Two tiers exist (Exploration and Design, and Development and 
Implementation) for each track. The tiers correspond to differences in scale and funding levels. 


NSF IUSE Program Tracks 


=taler=te[=ve| Institutional & 
Student Community 
MF Taaliale) Transformation 


Exploration Development Exploration Development 
& Design & Implementation & Design & Implementation 


Two Tiers Two Tiers 


The focus of the Engaged Student Learning track is on designing, developing, and implementing 
research studies on STEM learning models, approaches, and tools that show promise for 
increasing the engagement of undergraduate students in STEM learning and lead 
to measurable and lasting student learning gains. The expectation is that these research 
projects are evidence-based and build upon prior Knowledge about how people learn. 


Promoting Transformation of Undergraduate STEM Education Workshop Summary Report 


The focus of the Institutional and Community Transformation track is on increasing the propa- 
gation of highly effective methods of STEM teaching and learning. Institutions or collaborations 
across multiple institutions are supported to use innovative approaches to increase the use of 
evidence-based teaching and learning practices. The expectation is that these research projects 
have identified a problem to be addressed, goals to be achieved and strategies to meet the 
established goals, and that they include an identified theory of change that is guiding the work. 
The projects are expected to have mechanisms to both assess the effectiveness of the project 
and examine the process of change. 


The NSF’s Division of Undergraduate Education has made approximately 60 institutional awards 
beginning with those made through the Widening Implementation and Demonstration of Evi- 
dence Based Reforms (WIDER) program (NSF 13-552) and continuing through the Institution- 
al and Community Transformation IUSE program for the past four years (the current IUSE 
Solicitation is NSF 17-590). Change leaders, including leaders in funding agencies, need re- 
search-based guidance as they envision, organize, and facilitate major transformation efforts 
designed to advance change initiatives to improve the quality and effectiveness of undergrad- 
uate STEM education. 
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Description and Goals 


AAU hosted the Essential Questions and Measures: Assessing Institutional Transformation of 
Undergraduate STEM Education Workshop on February 6-7, 2019 in Washington, D.C. Twenty- 
seven experts involved in leading, researching, evaluating and funding efforts to transform 
undergraduate STEM teaching and learning practices were convened as participants. The 
workshop was facilitated by three AAU staff members, and six NSF staff members joined as 
observers (a list containing the names, institutional affiliations and titles of all workshop par- 
ticipants is found at the end of this report). 


The workshop was grounded and guided in part by what has been learned and documented 
in prior research. In particular, research and engagement in national conversations under- 
taken by AAU and its research partners Adrianna Kezar, James Fairweather, Linda Slakey, 
and Mary Deane Sorcinelli provided a foundation for the workshop (see, Association of 
American Universities, 2013, 2017a, 2017b; Kezar, 

2018; Miller, Fairweather, Slakey, Smith, & King, [s the intention of the 


2017). Research and project activities of the AAU . . . . 
Undergraduate STEM Education Initiative are cur- Investigation to examine 


rently supported by two active NSF awards (grant what were the outcomes 
numbers 1432766 and 1625532), and through the 
support of non-NSF awards—most notably major of the reform effort, or to 


grants from the Helmsley Charitable Trust, as wel @XxaQmine how the reform 
as the Northrop Grumman Foundation. In addition, . 

AAU previously received an NSF WIDER award effort was implemented? 
(grant number 1256221) which resulted in Essential 

Questions & Data Sources for Continuous Improvement of Undergraduate STEM Teaching 

and Learning (Association of American Universities, 2017a). That study provides a set of ques- 

tions that can be used at multiple levels within a university to assess progress along the set 

of key institutional elements identified in AAU’s Framework for Systemic Change in 
Undergraduate STEM Teaching and Learning (Association of American Universities, 2018). 


From this research emerges a Critical consideration when designing questions to examine, 
and indicators to document, transformations to undergraduate STEM teaching and learning. 
It may be asked as follows: Is the intention of the investigation to examine what were the out- 
comes of the reform effort, or to examine how the reform effort was implemented? Articulation 
of the focus of study is critical because the research questions to examine a project’s impact 
are different than the research questions to examine the change process of a project. 


Within each of these two lines of inquiry are a number of complexities. For example, the scope 
and scale-size of transformation projects (e.g., institutional, multi-institutional, or association- 
wide) present a challenge to both enact change and to study the change process (Association 
of American Universities, 2017b; Coleman, Smith, & Miller, in press; Kezar, 2018). Additionally, 
the types of relevant measures to evaluate classroom-based, department- and institution-level, 
and multi-institutional project effects vary greatly (Fairweather, Trapani, & Paulson, 2016). At 
the same time, higher education is a complex system and achieving transformation in under- 
graduate STEM education involves examining the relevance and impact of critical organiza- 
tion factors that can impede or facilitate progress (Austin, 2011; Kezar, Miller, Bernstein-Serra, 
& Holcombe, 2019; Miller, Fairweather, Slakey, Smith, & King, 2017). 
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Ultimately, understanding the issues and challenges in designing, leading, and researching 
institutional and multi-institutional transformation projects to improve undergraduate STEM 
education is of practical and theoretical importance since an increasing number of these proj- 
ects are underway. The workshop aimed to develop recommendations based on our collec- 
tive Knowledge in these areas that would enhance future solicitations for systemic institutional 
transformation projects. 


Activities 


The format of the workshop included a design thinking activity as well as small-group and 

whole-group discussions. These activities engaged participants in two key exercises: 

1. identifying essential questions and indicators of progress when studying the impact 
of a reform effort as compared to studying a reform effort’s process of change, consid- 
ering projects at multiple levels and; 

2. providing recommendations for enhancing future solicitations aimed at systematic 
transformation of undergraduate STEM teaching and learning building on what is known 
about fostering change. 


During the first segment of the workshop, participants worked together to identify import- 
ant questions, forms of evidence and indicators of progress that should be considered when 
examining the impact of an implementation project, as well as when examining the change 
process that occurred in an implementation effort. Throughout this exercise workshop par- 
ticipants were mindful of the multiple levels at which project work can occur. Although they 
focused on institutional-level and multi-institutional level transformation efforts, workshop 
participants were asked to consider additional levels such as efforts that comprise funded 
portfolios and nationwide efforts (see the matrix grid developed by AAU to frame the work- 
shop discussion and acknowledge existing research). 


Building off the information that was gathered and discussed, workshop participants then 
worked together to create recommendations to improve future funding solicitations for proj- 
ects aimed at systemic improvements to undergraduate STEM education. 
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Institutional 
Improvement 


RESEARCH PROJECT FOCUS 


Examining the Impact of 
an Implementation Effort 


What were the outcomes 
of the reform effort? 


Existing Resources: 

Within class faculty member 
research 

(National Research 

Council, 2012), cross- 
department or college 
evidence (e.g., institution- 
wide data analytics) 


Examining the 
Change Process of an 
Implementation Effort 


How was the reform 
effort implemented? 


Existing Resources: Support 
for High-Impact Practices 
(Kezar & Holcombe, 2017); 
Promoting Evidence-Based 
Change (Austin, 2011) 


Multi-institutional 
Projects/Networks 


Existing Resources: 

AAU Undergraduate STEM 
Education Initiative 5-Year 
Status Report (Association 
of American Universities, 


2017; Miller, et al., 2017) 


Existing Resources: 
Increasing Student 
Success in STEM: A Guide 
to Systemic Institutional 
Change (Elrod & Kezar, 
2016); Communities of 
Transformation and Their 
Work Scaling STEM Reform 
(Kezar & Gehrke, 2015); The 
Strategic Role of a National 
Organization (Kezar, 2018) 


Portfolio of Funding 
(e.g., NSF IUSE, HHMI 
Inclusive Excellence) 


National-Level 


Existing Resources: 
Indicators for Monitoring 
Undergraduate STEM 
Education (National 
Academies of Science, 
Engineering and Medicine, 
2018) 


Existing Resource: Achieving 
Systemic Change (Coalition 
for Reform of Undergraduate 
STEM Education, 2014) 
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Data Analysis 


Materials from small-group activities and notes from whole-group discussions were collected 
and analyzed. The analysis aimed to summarize information and to find patterns in the data 
that address the focus areas—factors related to essential questions and indicators for study- 
ing the impact of change and the process of change across multiple levels. The workshop 
design and analysis also generated suggestions for future solicitations to catalyze systemic 
transformation of undergraduate STEM teaching and learning. 


Using computer-assisted qualitative data analysis software, a coding process was carried 
out to capture variety in the responses and to find commonalities across all of the data. 
Descriptive codes were developed to summarize information. Then, similar codes were 
grouped into categories and subcategories. Codes, categories, and subcategories were 
refined and codified to form considerations and recommendations. In total, 20 consideration 
areas and two major solicitation recommendations emerged from the data. The complete 
set of considerations are listed and described in the appendix. Additionally, the appendix 
contains tables showing the results of data analyses in more detail. 
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Institutional and Multi-Institutional 


Projects to Transform 


Undergraduate STEM Education 


A critical component of the IUSE program is that it purposefully links the assessment of the 
stated goals and outcomes of the project with the comprehensive evaluation of the organi- 
zational change process. Such evaluations need to document successes and failures and 


identify critical elements necessary for change, all while 
considering the unique characteristics of the project 
goals and institutional contexts. 


Data from the workshop regarding factors that are 
important to consider when examining what were the 
outcomes of a reform effort and when examining how a 
reform effort was implemented, for both institutional and 
multi-institutional transformation projects, were analyzed 


A critical component of the 
IUSE program is that it 
purposefully links the 
assessment of the stated 

goals and outcomes of the 
project with the comprehensive 
evaluation of the organizational 


by looking at the frequency of responses. The top three 
considerations in each of the four areas are seen in the 


change process. 


matrix below. In the two instances where four categories are listed, examining the impact of 
institutional improvement efforts and examining the change process of multi-institutional 
efforts, there existed ties in counts for second and third places, respectively (see appendix 
data tables for more details). 


Looking at the similarities and differences of the top considerations appearing down the rows 
and across the columns of this matrix yields the following key findings: 


Conditions for sustained change appears in each of the cells of the matrix. This indi- 
cates the importance of understanding and taking into consideration the conditions under 
which transformations to improve undergraduate teaching and learning are sustained— 
for both institutional and multi-institutional change projects, and also when looking at 
questions about the impact, as well as the change process of efforts. 


Student and alumni learning outcomes appeared as an important consideration for 
examining the outcomes of institutional and multi-institutional transformation projects, 
but it was not a top consideration when examining the change process of projects. 


Faculty approaches to teaching and teaching improvement was a top consider- 
ation for studying the change process in institutional and multi-institutional efforts, as 
well as for examining the outcomes of institutional change projects. 


Not surprisingly, examining cross-institutional collaborations and peer networks 
was a top consideration for studying both the impact and change process of multi-insti- 
tutional efforts, but it was not applicable to studying the impact or change process of 
institutional improvement projects. 
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™ Some considerations appeared in only one quadrant of the matrix. Department 
commitment emerged as a top consideration in studying the impact of institutional 
reform efforts. On the same row, alignment of evidence-based practices with 
rewards, hiring and training appeared as a top consideration for examining the 
change process of institutional reform efforts. Lastly, common data was a top 
consideration for examining the change process of multi-institutional reform efforts. 


Examining the 
Impact/Outcomes 
of the Reform Effort 


Conditions for 
sustained change 


@ Student and alumni 
learning outcomes data 
Institutional 
Improvement @ Faculty approaches to 
teaching and teaching 
improvement 


@ Department commitment 


@ Conditions for 
sustained change 


@ Student and alumni 
learning outcomes data 


Multi- 
Institutional @ Examining cross-institutional 
Projects/ collaborations and 
Networks 


peer networks 


Examining the 
Change Process 
of the Reform Effort 


# Conditions for 
sustained change 


@ Faculty approaches to 
teaching and teaching 
improvement 


# Alignment of evidence- 
based practices with 
rewards, hiring and training 


@ Conditions for 
sustained change 


@ Faculty approaches to 
teaching and teaching 
improvement 


H Examining cross-institutional 
collaborations and 
peer networks 


Hi Common Data 
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Alignment of evidence-based practices with rewards, hiring and training: 
Alignment of the use of evidence-based practices with policies, practices and dis- 
cussions regarding faculty promotion and tenure, the evaluation of teaching, and 
recognition and rewards for exceptional teaching efforts. Additionally, this category 
includes the alignment of evidence-based practices with policies and practices 
around the hiring and development of new faculty and instructors, and the pedagog- 
ical training of graduate students. 


Common data: The importance of having common data sources and shared met- 
rics, data approaches and analyses within and across projects. 


Conditions for sustained change: Examining the conditions under which changes 
to improve undergraduate teaching and learning are sustained. These include look- 
ing at and understanding the kinds of leadership and clout; the types of structures, 
strategies, and supports; the necessary amounts of readiness; the kinds of informa- 
tion and the nature of motivations necessary to create systemic change. 


Department commitment: The existence of a commitment to and shared owner- 
ship of student success and excellence in teaching within the department. This can 
be evident though departmental culture, practices and discussions; department-wide 
curricula reform efforts such as course section alignment; and the collection and 
use of within-department metrics and student data to inform educational improve- 
ment efforts. 


Examining cross-institutional collaborations and peer networks: Examining 
various characteristics of cross-institutional collaborations and peer networks, includ- 
ing their formation, nature, longevity and internal and external supports; their bene- 
fits and outcomes; and inquiries around the spread of effective practices and les- 
sons learned across and beyond the networked institutions. 


Faculty approaches to teaching and teaching improvement: Looking at and 
understanding how faculty and instructors approach teaching, their engagement 
with developing their teaching, as well as the activities they do when they teach. This 
incorporates examining classroom practices; faculty and instructors’ attitudes and 
beliefs about teaching, including their desire to improve or change their teaching; 
faculty and instructors’ knowledge about, comfort with and buy-in to evidence-based 
teaching practices; their perceived sense of teaching support; their involvement in 
teaching professional development; and their engagement in educational research. 


Student and alumni learning outcomes data: Collecting and examining student 
learning, enrollment, experience, achievement and outcomes data. This category 
also includes collecting and examining learning and outcomes data of alumni. 
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Portfolio and National Level Work 
to Transform Undergraduate 
STEM Education 


At the portfolio level, the top considerations when examining the outcomes of reform efforts 
and the change processes of efforts were the same. They involved commitment to building 
upon what is already known; to understanding the conditions necessary to effect change, 
including the real implications of various institutional contexts; and to striving toward equity 
and inclusivity. Looking at the national level, the top considerations when examining the 
change process of efforts were similar. They included understanding the conditions needed 
to sustain change, as well as the indicators of progress. 


When examining the impact of reform efforts at a national level it is important to acknowl- 
edge that the National Academies of Sciences, Engineering and Medicine (2018) convened 
a consensus study to develop a set of national-level indicators to measure the status and 
quality of undergraduate STEM education over multiple years. This focus was important 
because current undergraduate STEM education reform initiatives tend to gather detailed, 
local data that are useful and appropriate for local feedback and improvement at the indi- 
vidual, departmental, institutional, or system level. However, such detailed data are not ade- 
quate for providing a broad, national picture of STEM teaching and learning. This limitation 
was reinforced by the workshop. The top considerations for demonstrating the impact of 
transformation on undergraduate STEM education at the national level were how it affected 
the public understanding and attitudes toward science and higher education, as well as its 
documented student learning and alumni outcomes data. 
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Recommendations for the NSF 
Improving Undergraduate STEM 
Education Program Solicitation 


Two main recommendations for refining future solicitations aimed at systemic transforma- 
tion of undergraduate STEM education emerged from the workshop. The solicitation recom- 
mendations align with the key findings discussed above and offer a strategy for ensuring 
future funded projects address these critical elements. 


The first recommendation is to provide more structure to the solicitation by explicitly articu- 
lating what is wanted and expected to be included in proposals, and by including detailed 
guidelines for successful proposals and successful proposal elements. The second is to 
increase the number of successful proposals submissions from more varied institutions and 
institutional types. Workshop participants provided concrete suggestions for ways to achieve 
the two recommendations, which can be seen in the two tables on the following page. 


In discussions about the first recommendation that future solicitations include more struc- 
ture and guidelines, workshop participants emphasized the importance of framing the call 
around systemic change, making clear the intention and goals of the solicitation, and being 
mindful of the language and attention to detail used in the call. Workshop participants said 
those improvements would make solicitations become better catalysts for change. 


Workshop participants also stressed the need for solicitations to incorporate specific 
requests to help ensure that projects successfully enact and sustain change. Suggestions 
included requiring proposals to demonstrate institutional buy-in and contain well thought- 
out plans for evaluation. To help submissions meet the conditions of the more deliberately 
structured call, workshop participants indicated the need for the solicitation to provide more 
detailed guidance on required proposal elements and to offer examples of past proposals 
that have been successfully awarded. 


When talking about the second recommendation to increase the number of successful pro- 
posal submissions from more varied institutions and institutional types, the analogy of 
“expanding the choir” emerged. Workshop participants noted that individuals from similar 
institution types, or in some cases from the same institutions, are most often principal inves- 
tigators or project team members on NSF IUSE grants. Given this, there is a need to build 
across a wider variety of institutions a capacity to be prepared to engage in work toward 
systemic improvements in undergraduate STEM teaching and learning. 


Workshop participants suggested several ways to scaffold readiness of institutions that are 
new to this work, as well as ways to support individuals who are submitting IUSE proposals 
for the first time. Some of the suggestions included offering planning grant or readiness grant 
opportunities to institutions beginning to engage in this work and providing resources, con- 
nections, and mentoring support to first-time proposal submitters. Additionally, it was sug- 
gested that mutually beneficial, cross-institutional capacity-building partnerships be encour- 
aged to allow successful awardee institutions and institutions from underrepresented 
institutional types to collaborate to enact change. 
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In addition to the two main recommendations for future IUSE solicitations, some workshop 
participants discussed other types of funding calls or awards that NSF could make available 
to advance specific areas of work related to fostering change, or to incubate creative proj- 
ects, such as “Idea Labs.” A few workshop participants also raised some important consid- 
erations related to the award selection process, including the need to identify the gaps of 
work in the portfolio, and the importance of educating and training proposal reviewers. 


Provide more structure, explicit wording and guidance in calls 
Include more detailed expectations and guidelines for a successful proposal 
and successful proposal elements; make clear the intention of the solicitation; 
pay particular attention to the language used in the call 


V Focus calls around change 

>» Focus calls on a system view of change; include an articulation of alignment of institutional 
vision, change theory, leadership, and activities 

> Explicitly fund research to understand and better implement processes to promote and 
spread change 

> Require proposals to specify change theories and vision of change, and tie proposed strategies 
and actions to them 

> Offer two distinct funding tracks: (1) Implementing, Scaling, or Transferring Change Track and 
(2) Studying, or Exploring Change Track 


Include specific asks in calls 

Require dissemination beyond publication, and the dissemination of both successes and failures 
Have proposals include an audit of existing efforts, needs, strengths and challenges 

Have proposals demonstrate connections with other institutional change projects 

Have proposals demonstrate information sharing, using and synthesis 

Have proposals include the use of institutional data 

Have proposals include the use of SoTL and DBER literature 


vVvvvvvvd 


Ensure proposals have institutional buy-in 

Require proposals to demonstrate connections with other individuals, leaders or units 
within the institution 

Ensure institutional engagement and that proposals are a team effort 

Ensure involvement of institutional leaders 

Provide institutional incentives for project Success 

Have institutional-level calls 


v< 


vvvv 


Include expectations for project evaluation and reporting 

Require proposals to include a map of measures related to goals and to include a formative 
assessment plan and reflective continuous improvement plan 

Provide guidance on expectations for internal and external evaluation 

Require annual reports that are formative and reflective 

Require the evaluator to be part of the project team 

Require plan for evaluating impact in terms of student inclusion 


v< 


vvvv 
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Increase the number of successful proposal submissions from 
more varied institutions and institutional types 
Structure calls to be more adaptable for submissions from a wide range of institutions 
and institutional types; for example, meet campuses where they are - have distinct 
expectations for projects at campuses that are experienced in this work, and for 
projects at campuses that are new to this work; recruit and support proposals from 
a wide range of institutions and institutional types 


Vv Provide resources, opportunities and connections to support successful proposal 
submissions from more institutions 

> Provide support, connections, mentoring, guidance, and/or a community for success for 
first-time proposers or those at institutions new to this work 

>» Offer assessment of readiness grant opportunities and/or planning grant opportunities; 

scaffold institutional readiness 

Include a list of resources and past awardees to contact in the solicitation 

Support the development of cross-institutional capacity-building partnerships 

Fund opportunities to apply what is known to be effective in new contexts 

Support opportunities for cross-institutional learning 

Identify lessons learned from projects in the portfolio, publish the lessons in open access 

spaces and include these references in the solicitation 

Offer two distinct funding tracks: (1) Readiness Track and (2) Traditional Track 


vVvvvvv 


Vv 
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Ann E. Austin, University Distinguished Professor 
of Higher, Adult, and Lifelong Education; Associate 
Dean for Research, College of Education; Assistant 
Provost for Faculty Develooment—Academic 
Career Paths, Michigan State University 


Over the past decade and a half, many efforts have focused on transforming STEM undergrad- 
uate education. Recently, some observers have used the term “tipping point” to suggest that 
the collective efforts are indeed making a difference—that how faculty teach, how institutions 
support evidence-based teaching, and how students learn are changing in noteworthy ways. 
The recent workshop on “Assessing Institutional Transformation of Undergraduate STEM 
Education,” hosted by the Association of American Universities, provides evidence of transfor- 
mation while, simultaneously, revealing the challenges and opportunities still to be addressed. 


The overall approach of the workshop planners to designing the meeting and articulating the 
guiding questions is testimony to the significant progress already achieved in collective under- 
standing of how systemic change occurs. First, the invitation to the workshop and the fram- 
ing of its agenda took a decidedly systems approach, recognizing that any discussion of mea- 
sures of impact or elements of change processes must take into account the multiple levels 
of the system in which STEM reform is occurring; that is, change occurs in departments, in 
institutions, within cross-institutional networks, and at the national level. Recognizing and orga- 
nizing change efforts within a systems perspective shows an important recognition of the 
nature of higher education institutions, an awareness of complexity which is an essential ele- 
ment in successful change efforts. Second, the workshop emphasized that “assessing trans- 
formation” requires attention both to the impact of implementation efforts and to the change 
processes themselves. Again, this framing advances the conversation in highlighting the rele- 
vant elements of understanding change. Third, a recognition of the importance of context in 
understanding and advancing change permeated the framing, facilitation, and conversations 
of the workshop. These features of the workshop show that efforts to transform undergradu- 
ate STEM education have reached a degree of maturity and sophistication, such that a work- 
shop bringing together national change leaders can be based on collective recognition that 
change is complex and challenging and requires a systemic approach. 


| also noticed that the workshop participants were eager to delve into the questions posed in 
the workshop, ready with examples, observations, and experiences to inform their discussions. 
By the conclusion of the workshop | had reached a compelling conclusion: Many efforts have 
been conceptualized, implemented, and, in some cases, funded. Progress in the project of 
transforming STEM education is note-worthy and encouraging, but the path ahead remains 
long and challenging. Even if a tipping point has been reached in which change efforts are 
enjoying demonstrable impacts (which | think has and is occurring), the time is right for the 
community of change leaders to step back, assess where we collectively have been and what 
has been achieved, articulate what has been learned about change in higher education, and 
clarify priorities for further attention. 


Continuing efforts to systematically reflect, analyze, prioritize, and frame the path forward, based 
on what has already been achieved, should address several key questions. What has worked 
and why in advancing transformation in undergraduate STEM education? What strategies and 
interventions have not worked, and why not? What has been learned about effective strategies 
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as they pertain to various target levels of transformation (department, institution, multi-institu- 
tional network, national)? How does context matter in efforts to foster transformation (a topic 
that was woven through the discussions in the workshop)? Around what issues has there been 
saturation in terms of projects and attention, and what issues, populations, or locations of trans- 
formation have been overlooked? Who has been involved in transformation efforts, and who is 
not fully participating in this agenda? What are the reasons some are not participants and are 
there ways to be more inclusive? What should be the next priorities for funding agencies, 
national associations, and institutional change leaders, and how might the efforts of these var- 
ious stakeholders be more fully and productively connected (and when would such connec- 
tions be valuable)? This workshop was a highly productive starting point in gathering stakehold- 
ers to wrestle with these questions. | hope opportunities for advancing consideration of these 
issues will continue. 


| can envision several practical and impactful steps forward to continue the synthesis, compar- 
ison, and deep discussion required to address these cross-cutting questions. One useful step 
already emerging in several conversations (for example, in the plans emerging in the National 
Academies’ Roundtable on Systemic Change in Undergraduate STEM Education) is to “map the 
andscape” of change efforts to reform STEM undergraduate education. This effort would involve 
inventorying initiatives (over the past decade, for example) designed to advance STEM reform 
efforts, noting the level of the system each initiative targets (i. e., course, department, institution, 
cross-institution, national, including funders’ priorities), and specific outcomes from each. Such 
a mapping process, building on project reports, could help identify parts of the system where 
efforts and progress have been extensive, and conversely, areas where more work is needed. 
Researchers, reform leaders, and funding agencies could benefit from this mapping. 


Closely related to a mapping process might be several facilitated national convenings, hosted 
by a funding agency or national association. These meetings could bring together stakehold- 
ers, Pls, and institutional leaders involved in the STEM reform movement to grapple with the 
compelling questions mentioned above and, through facilitated conversation, to identify and 
discuss lessons learned about advancing change goals. The National Science Foundation’s 
ADVANCE Institutional Transformation program, focused on increasing the recruitment, reten- 
tion, and success of women in STEM fields in higher education, provides an example of how 
PI meetings can be the base for developing active networks of institutional leaders and 
researchers. Over more than a decade, the ADVANCE PI meetings have cultivated active inter- 
actions among institutional change leaders who eagerly share lessons learned and innovative 
strategies for promoting change. 


A third idea, mentioned in the workshop, is to establish repositories for materials, instruments, 
and tools that have been produced in various funded projects to advance Undergraduate STEM 
Education. A curated repository (with each tool accompanied by information about its purpose, 
how it can be used, who has been using it, who has ownership, and the impact of the tool) 
would require dedicated curating. Such a repository could help jump-start the work of institu- 
tions new to systemic reform in STEM education and their leaders. A fourth strategy might be 
to provide leadership development opportunities focused on approaches and strategies for 
conceptualizing, designing, and implementing systemic undergraduate STEM reform at each 
level of the system—department, college, university, and storm. Department chairs, deans, 
provosts, and other senior leaders, as well as leaders of scholarly and professional associa- 
tions, need to have the tools to work with colleagues to implement STEM reform. Fifth, much 
is Known about what teaching strategies are most effective in fostering student engagement 
and learning. A project already under discussion, and offering much promise, is discussion 
across relevant stakeholders around professional standards in teaching. Each of these ideas 
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seems promising at this moment within the movement to transform undergraduate STEM edu- 
cation, and, in fact, several are in nascent or more developed stages of consideration within 
various groups, such as the National Academies of Sciences, Engineering, and Medicine, the 
AAU, and smaller research and leadership groups. 


In addition to these ideas for expanding a national network of change leaders and other stake- 
holders committed to assessing progress, highlighting lessons, and aiding higher education 
institutions in addressing challenges in institutional transformation, | want to emphasize two 
other issues discussed at the AAU workshop. First, participants offered specific and practical 
suggestions for revisions in the grant solicitation for the NSF IUSE Program. Ideas included 
requiring applicants for grants to provide an analysis of their institutional context, including bar- 
riers as well as existing programs with which they would coordinate; to discuss the several the- 
ories that inform their understanding of the problem they propose to address and their strat- 
egies to do so; to identify how project leaders are and will engage institutional leaders and 
other key stakeholders; and to discuss how evaluation plans would be integrated into the 
change agenda and would explicitly inform project development and implementation. As a 
researcher with grant experience, former NSF Program Officer, and institutional leader, | find 
these ideas very useful and hope they will be considered. Second, workshop participants 
observed that some institutions are less poised than others to compete successfully in the 
competition for funding to promote institutional change. | agree, and support the ideas raised 
in the workshop for scaffolding institutions less experienced in the funding arena. Some of 
these ideas included providing workshops and “ideas labs” in which institutional leaders could 
incubate ideas and get feedback, encouraging proposals that adapt ideas developed in one 
institutional context for another institutional type, and compiling and disseminating resources 
for those new to developing proposals for institutional transformation. 


| appreciate the workshop that AAU hosted, as well as the energetic participation of each 
attendee. | sense that we are indeed at a “tipping point” where the efforts to transform under- 
graduate STEM education are gaining traction and taking hold with faculty, in classrooms, 
and across institutions. By providing opportunities for those working to effect systemic change 
in higher education to reflect and compare experiences, convenings like the AAU workshop 
help leverage research findings and practical experiences and encourage continued progress 
and innovation. 
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Karl A. Smith, Professor Emeritus of Civil, 
Environmental, and Geo- Engineering, University 
of Minnesota and Cooperative Learning Professor 
of Engineering Education, Purdue University 


The current AAU Undergraduate STEM Education Initiative to influence the culture of STEM 
departments so faculty members are encouraged and supported to use evidence-based teach- 
ing practices as well as assess the transformation of undergraduate STEM education is import- 
ant and timely. As part of this initiative AAU engaged scholars and institutional leaders in a dis- 
cussion about the National Science Foundation Improving Undergraduate STEM Education 
solicitation and more broadly the state of institutional transformation in undergraduate teach- 
ing and learning. | was pleased to engage in this workshop and encourage the community to 
invest in continual assessment of the transformation, since periodic assessments are not suf- 
ficient to facilitate change. 


I’ve been at the University of Minnesota since 1972 when | started a position in an engineer- 
ing research laboratory. | was encouraged to complete a PhD and explored a traditional engi- 
neering PhD. However, in the mid-70s | switched from a traditional engineering PhD program 
to an educational psychology PhD program. The was due in large part to an experience | had 
in a teaching assignment where the approach that had been used on me (lecture-home- 
work-exams) failed and | thought there had to be a better way to do it. | searched for other 
ways to help students learn and discovered the Johnson & Johnson cooperative learning 
model. | implemented it in my classes, systematically researched it, and in 1981 introduced it 
to the engineering education community.” In the subsequent years |’ve been involved in 
numerous initiatives to encourage faculty to embrace evidence-based instructional practices, 
especially cooperative learning, and | highlight three experiences at the University of Minnesota, 
Michigan State University, and Purdue University. 


The University of Minnesota has been involved in many, mostly externally grant-funded, proj- 
ects to improve student learning. For example, in the early-80s the University engaged in the 
Northwest Area Program on Active Learning. Steve Schomberg, project director, noted that 
“Over a three-year period (1982-1985), they tried out 20 methods of involving students in 
instructional activities, and trained more than 200 faculty to use one or more active learning 
strategies.”* Subsequent projects have emphasized engaging students in large classes, inter- 
nationalizing the curriculum, preparing future faculty, and so forth. Although there has been a 
sustained effort at the University of Minnesota to embrace evidence-based instructional prac- 
tices, especially interactive learning, there is little evidence of widespread transformation in 
STEM disciplines. Biology is the exception with the implementation of the SCALE-UP model 
and a dedicated building with active learning classrooms. Also, A Guide to Teaching in the 
Active Learning Classroom: History, Research and Practice was published.* 
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Michigan State University was selected to participate in the Lilly Endowment Teaching Fellows 
program in 1991 and | was invited to serve as a consultant to the program. The impact of the 
Lilly Endowment Teaching Fellows program was documented by Ann Austin.® In 1993 President 
Peter McPherson introduced the Six Guiding Principles as part of an initiative to consider what 
it means to be a Land Grant University in the 21st Century. Guiding principle number two is 
Achieve More ACTIVE LEARNING. | was invited to serve as a Senior Consultant to the Provost 
and my role was to help implement cooperative learning at Michigan State. Today, there are 
many pockets at Michigan State where evidence-based practices, such as cooperative learn- 
ing, are implemented. 


Purdue University launched one of the first engineering education departments and PhD pro- 
grams in 2005. | was on sabbatical at Purdue during the 2004-2005 academic year and par- 
ticipated in conversations about the creation of a department. In 2006 | began phased-retire- 
ment from the University of Minnesota and joined the Purdue Engineering Education department 
part-time to help create the PhD program. The PhD program didn’t follow the traditional Johns 
Hopkins model but instead embraced the findings of studies of the doctorate (Carnegie Initiative 
on the Doctorate, Woodrow Wilson Responsive PhD, and Re-envisioning the PhD). The PhD 
program is competency-based with ten competencies, portfolio-guided assessment, and 
embraces approaches such as a readiness assessment instead of the traditional qualifying 
exam. Recently Purdue Engineering Education graduated the 100th PhD and | am hopeful that 
these and many other Engineering Education and STEM DBER graduates can help lead the 
transformation of STEM undergraduate education. 


These experiences not only inform but also shape my reflection on the workshop hosted by 
AAU. The National Science Foundation has had an enormous influence on the advance of evi- 
dence-based practices in STEM undergraduate education, for example through the Engineering 
Education Coalitions, and funding CCLI and IUSE projects. The 1996 report to the National 
Science Foundation — Shaping the Future: New Expectations for Undergraduate Education in 
Science, Mathematics, Engineering and Technology®-articulated ambitious goals for SME&T 
education and highlighted the importance of inquiry (p. ii): 

All students have access to supportive, excellent undergraduate education in science, 

mathematics, engineering, and technology, and all students learn these subjects by 

direct experience with the methods and processes of inquiry. 


The authors recommend that inquiry be placed at the core of learning goals consistent with 
ABET and they have an emphatic recommendation that SME&T faculty (p. iv): 
Believe and affirm that every student can learn, and model good practices that increase 
learning; starting with the student’s experience, but have high expectations within a 
supportive climate; and build inquiry, a sense of wonder and the excitement of discov- 
ery, plus communication and teamwork, critical thinking, and life-long learning skills into 
learning experiences. 


The emphasis on institutional and community transformation is critical to sustaining and scal- 
ing change. The predominant emphasis on individual faculty and classes has had some impact; 
however, has not led to the extent or quality of implementation of evidence-based instructional 
practices that would benefit many students. The three examples | cited from my experience at 
the University of Minnesota, Michigan State University and Purdue University, support the idea 
of the importance of institutional leadership. 


5 Ann E. Austin, “Supporting Junior Faculty Through a Teaching Fellows Program,” in Developing New and Junior Faculty: New 
Directions for Teaching and Learning No. 50, eds. M.D. Sorcinelli and A.E. Austin (San Francisco: Jossey-Bass, 1992), 73-86. 

6 Shaping the Future: New Expectations for Undergraduate Education in Science, Mathematics, Engineering, and Technology 
(Washington: National Science Foundation NSF 96-139, 1996). 
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Individual change is occurring and has been occurring for some time. In 2000 Jean MacGregor, 
Jim Cooper, Pat Robinson and | reported a synthesis of interviews with forty-eight individuals 
from across the United States who were infusing their large classes with small-group activities or 
were working explicitly to create student communities within large classes.’ One of the surprises 
from our interviews was the large proportion who noted that they thought they were the only one 
who were engaging their students in large classes. The sense of isolation persists to this day. 


The implementation of evidence-based instructional practices by necessity takes place at the 
individual faculty and classroom level and hence this is a critical area for further investigation. 
Faculty hiring, training and rewards has an effect and merits further study. The effect of the long 
socialization in conventional practice (lecture-homework-exams) likely makes it difficult for some 
faculty to imagine other ways of helping students learn. One indication of this is that evi- 
dence-based instructional practices such as cooperative learning have been gaining presence 
in STEM education; however, they are far from being the dominant practice, which continues 
to be lecture.® Mediating the socialization effect merits further investigation. Faculty beliefs about 
teaching and learning have been the subject of limited study, and deeper understanding of fac- 
ulty beliefs would likely reveal challenges and barriers that need to be addressed. 


The question of how much structure and direction to provide in solicitations for proposals is 
tricky as too much stifles creativity and too little results in widely disparate proposal submis- 
sions. My response is one of my favorites, “it depends.”® Perhaps a range is called for. For some 
clearly identified wicked problems, such as achieving more high fidelity implementation of evi- 
dence-based instructional practices, a highly structured format may be appropriate. For emerg- 
ing opportunity areas, such as embracing the findings from neurological research, a less struc- 
tured approach may yield more exciting results. 


My experience over the past six years with the NSF-funded I-Corps™ for Learning (Il-Corpos™ 
L) project, which was focused on adapting the Lean Startup model (Search for a sustainable 
and scalable model — customer discovery — agile engineering (iterate and increment)) to sus- 
taining and scaling education innovations, convinces me that it is essential to incorporate 
emphasis in proposals on sustaining and scaling innovations after the project funding ends. 


Continuing to build and expand the research community and especially broaden the landscape 
of those submitting competitive proposals is another thorny problem. Our experience helping 
build the engineering education research community indicates that it is possible; however, takes 
time and a concerted effort.10111218 


7 Jean MacGregor, James L. Cooper, Karl A. Smith, and Pamela Robinson, eds. Strategies for Energizing Large Classes: From 
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Jacqueline R. Roberts, Professor and Chair of 
Chemistry and Biochemistry, DePauw University 


The recommendations made in this report are valuable and an accurate summary of the 
“Essential Questions and Measures: Assessing Institutional Transformation of Undergraduate 
STEM Education Workshop” hosted by the Association of American Universities (AAU) this past 
February in Washington, DC. From the context of one small liberal arts college, this document 
provides a reflection on the following report recommendations: 

= Examining the key findings from the impact/outcomes of the reform effort 

=» Examining the key findings from the change process of the reform effort 

= Examining the IUSE solicitation recommendations 


Overall, much of the report resonates within the context of liberal arts schools. However, there 
are some aspects that may present challenges to smaller schools, while other aspects will ben- 
efit and potentially provide liberal arts schools with a clearer path to funding from the NSF 
Improving Undergraduate STEM Education (IUSE) program. 


Key Findings 

At the beginning of the workshop we were asked to identify essential indicators and factors 
when looking at implementation projects. The data collected from this activity were analyzed 
by AAU and reported as Key Findings for Institutional and Multi-Institutional Projects to 
Transform Undergraduate STEM Education (examining both the impact/outcomes of the reform 
effort and the change process of the reform effort). In writing this reflection, some of top cate- 
gories were reviewed (and in some cases grouped together) to provide a context from one small 
liberal arts school’s perspective. 


Common data: 

Student and alumni learning outcomes data: 

Having a common data source, shared metrics, and assessment strategies would be a tre- 
mendous help to liberal arts schools, as many are less likely to have a dedicated assessment 
specialist, normally do not have science/math education departments, and may only have one 
person in the office of institutional research to help with data analysis. In addition, having 
assessment tools available to collect student and alumni outcomes data would be beneficial 
to liberal arts colleges given their more limited resources. 


Alignment of evidence-based practices with rewards, hiring and training: 

Faculty approaches to teaching and teaching improvement: 

Conditions for sustained change: 

Alignment of evidence-based practices with rewards, hiring and training is key to institutional 
transformation. It is also important that faculty are constantly monitoring and updating their 
teaching to be more inclusive and focused on evidence-based practices. There are often bar- 
riers to this ideal process at liberal arts colleges compared to large research institutions. At the 
former there are much smaller science and math departments (in some cases, the entire depart- 
ment may have 3 faculty) which means that one recalcitrant faculty member can derail any 
reform efforts, so getting broader faculty “buy-in” is important at smaller schools. There is also 
a great deal of faculty autonomy at smaller schools, often with each faculty member being 
allowed to determine his or her own approach to a course, select a textbook, activities, and 
assessments. The upshot is there may be a lack of clear, shared departmental learning goals 
for individual courses or majors. In addition, once a faculty member is fully promoted with 
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tenure, in many cases there are no further personnel reviews, making it a challenge to convince 
or compel senior faculty to alter their course content or pedagogy. Although there are benefits 
of having greater faculty independence and this stronger “bottom up” structure at small liberal 
arts schools, this can present an impediment to motivating and organizing departmental change 
compared to at larger schools where there may be clearer hierarchical structure with “top- 
down” organization, which makes it easier to mandate shared learning goals, common labs, 
assessments, and pedagogies. On the positive side, when an entire department at a small lib- 
eral arts school does unite toward a common goal, a powerful transformation can be seen. 


Departmental commitment: 

The report emphasizes the importance of departmental commitment; however, as science and 
math departments tend to be small at liberal arts colleges, it might instead be more produc- 
tive to have science/math divisions as a whole commit to shared student successes. 


Examining cross-institutional collaborations and peer networks: 

In addition to suggestions in the report, cross-institutional collaborations and peer networks 
would also be beneficial to faculty at liberal arts colleges, especially those in smaller depart- 
ments. For example, a school’s consortium (e.g., Great Lakes Colleges Association (GLCA) or 
Associated Colleges of the Midwest (ACM)) could help coordinate partner schools in learning 
about evidence-based teaching and successful institutional change so that faculty in similar 
situations at other institutions could work together, thus reducing feelings of isolation and the 
unproductive “reinvention of the wheel.” 


IUSE Solicitation Recommendations 

Based on conversations during the workshop, two different recommendations for refining the 
solicitation for the IUSE application were developed. The first recommendation was to provide 
more structure, explicit wording and guidance for proposal calls. In addition to specific infor- 
mation and language to be used, it was suggested that NSF offer two distinct funding tracks: 
1) Implementing, Scaling, or Transferring Change, and 2) Studying, or Exploring Change. More 
explicit wording and guidance would help faculty at small liberal arts schools, as they may lack 
the infrastructure to support grant applications. For example, with regard to the recommenda- 
tion that dissemination beyond publication be required: Could previous awardees host webi- 
nars on the successes and failures of their project so these “lessons learned” could be trans- 
ferred to other campuses? Could consortia for small schools (e.g., the GLCA or the ACM) apply 
to the IUSE program to host centralized workshops? These could be themed workshops around 
one topic (e.g., Computer Science or Physics) or more general for all science/math depart- 
ments with different focuses based on the readiness of the school participating. Having a con- 
sortium of departments working together may help smaller departments or those with reluc- 
tant or disinterested faculty members. 


The second recommendation to NSF was to increase the number of successful proposal sub- 
missions from more varied institutions and institutional tyoes, and to also offer two different 
funding tracks: 1) Readiness, and 2) Traditional tracks. The addition of this “Readiness” track 
could help liberal arts schools apply for funds so the institution could develop the assessment 
strategy needed to collect baseline data, or to partner with other institutions who have received 
funding, or even help train faculty change agents for institutional reform. This recommendation 
would allow for smaller schools to start down the path for NSF funding via a readiness grant 
and potentially provide the mechanism by which they could apply for a more traditional grant. 
This would open up opportunities for new institutions to receive funding, so the same larger 
institutions do not continue to receive multiple grants while many smaller schools struggle with 
just trying to submit a proposal. 


Promoting Transformation of Undergraduate STEM Education Workshop Summary Report 27 


Michael J. Pullin, Associate Dean of Academic 
Affairs, Queensborough Community College, 
City University of New York 


As someone who has been a PI for NSF STEM education grants at both 2-year and 4-year insti- 
tutions, | am pleased with the recommendations detailed in this report. In particular, | think that 
the call for efforts to encourage proposals from a wider range of institutions is much needed. 


In 2014, after serving as a faculty member at a research university for 10 years, | moved to a 
2-year college as an administrator. One motivation for the move was my work in preparing 
STEM community college students for transfer to 4-year institutions and the pursuit of careers 
in the sciences and engineering. That work included REU and EPSCoR-funded summer 
research programs aimed at community college students, a STEP grant to prepare students 
for transfer, and a sabbatical semester spent teaching at a tribal 2-year college. Those expe- 
riences taught me that community colleges are home to many bright students with the talent 
and drive to successfully pursue STEM graduate degrees and research careers, but some- 
times without the guidance and support to do so. 


Many have correctly pointed out that increasing the recruitment, retention, and graduation of 
community college students in STEM degrees would help increase the number of underrepre- 
sented minority and first-generation college students who pursue STEM bachelor’s and grad- 
uate degrees. That would help correct the persistent and troubling underrepresentation prob- 
lem in many STEM fields and support the nation’s STEM workforce needs. 


Given the open admissions policies and high proportion of students needing developmental 
mathematics courses at community colleges, it is perhaps fair to ask if community college stu- 
dents can successfully pursue bachelor’s and graduate degrees in STEM fields. A National 
Center for Science and Engineering Statistics study of the academic background of science, 
engineering and health graduates found that about half of bachelor’s and graduate degree grad- 
uates started in community college.'* Additionally, students who earn an associate’s degree 
before transfer typically match (and sometimes outperform) students at the transfer institution 
who started there as first-time freshmen. Clearly, some STEM community college students are 
succeeding to earn higher-level degrees. 


However, many 2-year college students face difficult challenges. Community colleges are home 
to many low-income students who need to work to support themselves and may only be able 
to attend part time. They are also home to many first-generation students who may not have 
guidance from family and friends on how to navigate college and academic degree programs. 
They may also not understand the career and degree options in STEM. Additionally, many com- 
munity college students arrive underprepared. A significant fraction are required to take devel- 
opmental (non-credit, preparatory) mathematics, a significant barrier for students pursuing 
STEM degrees. Only a small fraction of 2-year college students arrive ready for calculus. 


More community college STEM students will succeed only when they are provided with monetary 
and academic support, advising and mentoring, and professional development (including research 
opportunities). Most importantly, they must have outstanding professors who utilize evidence-based 
teaching practices that increase student learning and performance. The IUSE program is designed 


14 Geraldine M. Mooney and Daniel J. Foley, “Community Colleges: Playing and Important Role in the Education of Science, 
Engineering, and Health Graduates,” National Center for Science and Engineering Statistics Info Brief, NSF 11-317 (2011): 1-5. 
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to support the development, implementation, and adoption of such teaching practices. Other NSF 
programs also fund a wide range of support programs for STEM students. 


Community college faculty hoping to submit proposals to the NSF face many challenges. Two- 
year college faculty are typically not expected to maintain an externally-funded research pro- 
gram. Accordingly, they have relatively high teaching loads, typically 12-15 credits per semes- 
ter. That leaves little time for grant writing or conducing funded research. Additionally, they may 
not have much knowledge of educational theory or findings and trends in STEM education 
research and little time to become knowledgeable. Many of the Pls for IUSE proposals today 
are STEM education specialists or content area faculty that have many years of progressive 
experience in building and running STEM education and outreach programs. For these rea- 
sons, it is difficult for a typical community college faculty member to write an IUSE proposal 
that competes successfully for funding. 


How do we help community college STEM faculty develop proposals that will compete suc- 
cessfully for funding and conduct educational research that adds significantly to the field while 
also increasing the number and success rates of their students? How can we help them cat- 
alyze change across their departments or institutions? The recommendations that resulted from 
this workshop are an excellent place to start. Personally, | think that providing community col- 
lege faculty interested in STEM education research with workshops and meetings, professional 
mentoring, and networks would be especially helpful. The greater inclusion of community col- 
leges in IUSE funded studies, along with faculty collaborators at those colleges, could also have 
a positive impact. 


Increasing the participation of 2-year colleges and their faculty in the IUSE program will not be 
easy. However, doing so will bring a broader range of perspectives to the program. It will also 
mean that many bright and promising students will find their place in the STEM endeavor. Their 
work will lead to scientific research that better represents their communities, improves their stand- 
ing in the world, and benefits everyone. | hope we can make that happen in the coming years. 
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Appendix 


Workshop participants worked together to identify important questions, forms of evidence and 
indicators of progress that should be considered when examining the impact of an implemen- 
tation project, as well as when examining the change process that occurred in an implementa- 
tion effort across the multiple levels at which project work can occur. In total, 20 consideration 
areas emerged from the data. They are listed alphabetically and described below. This list is 
followed by eight tables showing the results of the analysis of frequencies of consideration areas 
by examination focus (impact or change process) for each of the four project levels. 


Alignment of evidence-based practices with rewards, hiring and training: Alignment 
of the use of evidence-based practices with policies, practices and discussion 
of faculty promotion and tenure; evaluation, recognition and rewards for teaching; hiring 
and developing new faculty and instructors; and graduate program training. 


Alignment of student success with prestige: Alignment of student success with pres- 
tige and elevation of the teaching profession; creating professional standards of 
teaching; considering the negative impact of research culture. 


Broader Impacts: |mproved teaching-focused questions and projects coming from Broader 
Impacts sections of proposals; better monitoring of Broader Impacts outcomes. 


Collective vision: Existence of a collective vision, shared responsibility and culture that val- 
ues student success and excellence in teaching. 


Common data: Having common data sources, shared measures, metrics, data approaches 
and analyses. 


Conditions for sustained change: Examining conditions such as the leadership, clout, struc- 
tures, strategies, supports, readiness, motivation and data necessary to create systemic change. 


Department commitment: Department commitment to and shared ownership of student 
success and excellence in teaching, evident though practices, culture, discussions, Curric- 
ula reform, section alignment, metrics and use of student data. 


Examining cross-institutional collaborations and peer networks: Looking at the for- 
mation, nature, benefits, outcomes, support, longevity, and spread of lessons and practices 
of cross-institutional collaboration and peer networks. 


Existence of within-institution cross-boundary partnerships: Within-institution 
cross-boundary partnerships across disciplines, departments and units; spread of lessons 
and practices across disciplines, departments and units. 


Faculty approaches to teaching and teaching improvement: Examining classroom prac- 
tices; faculty attitudes and beliefs about teaching; faculty desire to improve or change teach- 
ing; faculty knowledge, comfort and buy-in to evidence-based practices; perceived sense of 
teaching support; faculty engagement in professional development and research 
on teaching. 
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Federal support, resources and policy: Existence of more supportive federal resources, 
structures, and policy discussions to advance work. 


Importance of context: |mportance of considering context in implementation, evaluation 
and findings. 


Indicators of progress: Indicators of progress, continual improvement, and saturation; new 
baseline standards. 


Institutional commitment: |nstitutional commitment to student Success and excellence in 
teaching; providing institutional support, resources and incentives; demonstrating institu- 
tional buy-in and prioritization. 


Professional certifications and endorsements: |nvolvement and endorsement by disci- 
pline-based professional societies and accrediting bodies; developing prestigious 
certifications. 


Public understanding and attitudes toward science and higher education: Public 
views and values of science and higher education; measures of scientific literacy; examin- 
ing best ways of communicating science to the public. 


Repositories to share materials and information: Creation of infrastructure to widely 
share materials, resources and information. 


Strive toward equity and inclusivity: Support diversity and equal opportunity for stu- 
dents, aspiring faculty and administrators, and institution types. 


Students and alumni learning outcomes data: Student learning, enrollment, experience, 
achievement and outcomes data; alumni learning and outcomes data. 


Take stock of what is known: Synthesize knowledge from project evaluations and research 
studies; conduct cluster analyses and landscape maps; identify strategies to address com- 
mon challenges. 
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Table 1 


Examining the Impact of Institution-wide Efforts 


Consideration Count Percent 
Student and alumni learning outcomes data 25 19.2 
Conditions for sustained change 19 14.6 
Department commitment 19 14.6 
Faculty approaches to teaching and teaching improvement 16 12.3 
Institutional commitment 1 8.5 
Alignment of evidence-based practices with rewards, hiring and training 9 6.9 
Indicators of progress 8 6.2 
Strive toward equity and inclusivity 8 6.2 
Collective vision 6 4.6 
Existence of within-institution cross-boundary partnerships* 5 3.8 
Importance of context 3 2.3 
Alignment of student success with prestige 1 0.8 
Common data 0 0.0 
Examining cross-institutional collaborations and peer networks O 0.0 
Repositories to share materials and information O 0.0 
Take stock of what is known O 0.0 
Total 130 100.0 
Note. *Category unique to institutional-level impact data 
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Table 2 


Examining the Change Process of Institution-wide Efforts 
Consideration 


Faculty approaches to teaching and teaching improvement 


Alignment of evidence-based practices with rewards, hiring and training 


Conditions for sustained change 

Department commitment 

Institutional commitment 

Collective vision 

Student and alumni learning outcomes data 

Indicators of progress 

Importance of context 

Alignment of student success with prestige 

Common data 

Repositories to share materials and information 

Strive toward equity and inclusivity 

Take stock of what is known 

Examining cross-institutional collaborations and peer networks 
Existence of within-institution cross-boundary partnerships 
Total 


Count 


19 


86 


Percent 
22.1 
20.9 
10.5 

8.1 
8.1 
7.0 
7.0 
5.8 
47 

1.2 

1.2 

1.2 

1.2 

1.2 

0.0 
0.0 
100.0 
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Table 3 


Examining the Impact of Multi-Institutional Efforts 
Consideration 

Examining cross-institutional collaborations and peer networks 
Student and alumni learning outcomes data 

Conditions for sustained change 

Common data 

Faculty approaches to teaching and teaching improvement 
Importance of context 

Indicators of progress 

Repositories to share materials and information 

Alignment of student success with prestige 

Collective vision 

Department commitment 

Strive toward equity and inclusivity 


Take stock of what is known 


Alignment of evidence-based practices with rewards, hiring and training 


Institutional commitment 


Total 


Count 


Oo wo woo wo ff DD OW 


4 4 


re 


37 


Percent 
21.6 
16.2 
10.8 

8.1 
8.1 
8.1 
8.1 
5.4 
27 
27 
27 
27 
27 
0.0 
0.0 
100.0 
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Table 4 


Examining the Change Process of Multi-Institutional Efforts 


Consideration Count 
Examining cross-institutional collaborations and peer networks 19 
Faculty approaches to teaching and teaching improvement 5 
Common data 4 
Conditions for sustained change 4 
Institutional commitment 3 
Student and alumni learning outcomes data 3 
Take stock of what is known 3 
Collective vision 2 
Department commitment 2 
Importance of context 2 
Indicators of progress 2 
Alignment of evidence-based practices with rewards, hiring and training 1 
Alignment of student success with prestige O 
Repositories to share materials and information O 
Strive toward equity and inclusivity O 
Total 50 


Percent 
38.0 
10.0 

8.0 
8.0 
6.0 
6.0 
6.0 
4.0 
4.0 
4.0 
4.0 
2.0 
0.0 
0.0 
0.0 
100.0 
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Table 5 


Examining the Impact of Efforts within a Funding Portfolio 


Consideration Count Percent 
Take stock of what is known 7 17.5 
Importance of context 6 15.0 
Conditions for sustained change 4 10.0 
Strive toward equity and inclusivity 4 10.0 
Department commitment 3 L5 
Indicators of progress 3 7.5 
Broader Impacts* 2 5.0 
Examining cross-institutional collaborations and peer networks 2 5.0 
Faculty approaches to teaching and teaching improvement 2 5.0 
Institutional commitment 2 5.0 
Repositories to share materials and information 2 5.0 
Student learning and alumni outcomes data 2 5.0 
Alignment of student success with prestige 1 2.5 
Alignment of evidence-based practices with rewards, hiring and training O 0.0 
Collective vision O 0.0 
Common data 0 0.0 
Total 40 100.0 
Note. *Category unique to portfolio-level data 
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Table 6 


Examining the Change Process of Efforts within a Funding Portfolio 
Consideration Count 
Strive toward equity and inclusivity 

Take stock of what is known 


Conditions for sustained change 


8 

8 

7 
Common data 2 
Department commitment 2 
Importance of context 2 
Indicators of progress 2 
Alignment of student success with prestige 
Broader Impacts* 
Institutional commitment 


Repositories to share materials and information 


Student learning and alumni outcomes data 
Alignment of evidence-based practices with rewards, hiring and training 
Collective vision 


Examining cross-institutional collaborations and peer networks 


Oe 1 © 


Faculty approaches to teaching and teaching improvement 
Total 36 


Percent 
22.2 
22.2 
19.4 

5.6 
5.6 
5.6 
5.6 
2.8 
2.8 
2.8 
2.8 
2.8 
0.0 
0.0 
0.0 
0.0 
100.0 


Note. *Category unique to portfolio-level data 
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Table 7 


Examining the Impact of National Efforts 


Consideration 


Public understanding and attitudes toward science and higher education* 


Student learning and alumni outcomes data 
Professional certifications and endorsements* 


Collective vision 


Alignment of evidence-based practices with rewards, hiring and training 


Alignment of student success with prestige 

Repositories to share materials and information 

Indicators of progress 

Conditions for sustained change 

Faculty approaches to teaching and teaching improvement 
Federal support, resources and policy* 

Institutional commitment 

Strive toward equity and inclusivity 

Department commitment 

Importance of context 

Take stock of what is known 

Common data 

Examining cross-institutional collaborations and peer networks 


Total 


Oo MO NM NY NY WO Ff FH HF OT OW 


a 


58 


Percent 
20.7 
13.8 

8.6 
8.6 
6.9 
6.9 
6.9 
5.2 
3.4 
3.4 
3.4 
3.4 
3.4 
1.7 
1.7 
1.7 
0.0 
0.0 
100.0 


Note. “Category unique to national-level data 
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Table 8 


Examining the Change Process of National Efforts 
Consideration 

Conditions for sustained change 

Take stock of what is known 

Indicators of progress 

Alignment of evidence-based practices with rewards, hiring and training 
Professional certifications and endorsements* 

Common data 

Importance of context 

Public understanding and attitudes toward science and higher education* 
Department commitment 

Examining cross-institutional collaborations and peer networks 
Federal support, resources and policy* 

Student learning and alumni outcomes data 

Alignment of student success with prestige 

Collective vision 

Faculty approaches to teaching and teaching improvement 
Institutional commitment 

Repositories to share materials and information 

Strive toward equity and inclusivity 

Total 


Count 


5 
4 
3 
2 
2 
2 
2 
2 


—_k 


CoO 0 OO Oo 


26 


Percent 
19.2 
15.4 
11.5 

77 
ea 
if 
th 
77 
3.8 
3.8 
3.8 
3.8 
0.0 
0.0 
0.0 
0.0 
0.0 
0.0 
100.0 


Note. “Category unique to national-level data 
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